2026-09-21 06:41:06,900 [ 328922 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2026-09-21 06:41:06,900 [ 328922 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2026-09-21 06:41:06,900 [ 328922 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2026-09-21 06:41:06,900 [ 328922 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_vegcn5 --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1]' test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards test_part_uuid/test.py::test_part_uuid test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple test_partition/test.py::test_system_detached_parts test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_relative_filepath/test.py::test_filepath test_reload_client_certificate/test.py::test_correct_cn_cert test_reload_client_certificate/test.py::test_wrong_cn_cert test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_s3_imds/test_session_token.py::test_credentials_from_metadata test_s3_imds/test_simple.py::test_credentials_from_metadata test_s3_latency_log/test.py::test_latency_log test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 test_s3_storage_class/test.py::test_s3_storage_class_right 'test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3]' test_s3_style_link/test.py::test_s3_table_functions test_s3_style_link/test.py::test_s3_table_functions_line_as_string test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] test_partition/test.py::test_attach_check_all_parts test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] test_recovery_replica/test.py::test_choose_source_replica test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] test_replication_credentials/test.py::test_credentials_and_no_credentials test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_reloading_storage_configuration/test.py::test_add_disk test_rocksdb_options/test.py::test_invalid_block_based_table_options [gw6] [ 1%] PASSED test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part [gw9] [ 2%] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 [gw6] [ 3%] PASSED test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists [gw9] [ 4%] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 [gw9] [ 5%] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 [gw6] [ 6%] PASSED test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts [gw0] [ 7%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw2] [ 8%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys [gw3] [ 9%] PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options [gw5] [ 10%] PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials [gw0] [ 11%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] [gw4] [ 12%] PASSED test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy [gw6] [ 13%] PASSED test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached [gw2] [ 14%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces [gw7] [ 15%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] [gw0] [ 16%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw8] [ 17%] PASSED test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery [gw5] [ 18%] PASSED test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials [gw0] [ 19%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] [gw6] [ 20%] PASSED test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex [gw4] [ 21%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config [gw2] [ 22%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys [gw5] [ 23%] PASSED test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials [gw0] [ 24%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] [gw7] [ 25%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] [gw6] [ 26%] PASSED test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple [gw8] [ 27%] PASSED test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata [gw3] [ 28%] PASSED test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options [gw0] [ 29%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] [gw6] [ 30%] PASSED test_partition/test.py::test_partition_simple [gw5] [ 31%] PASSED test_replication_credentials/test.py::test_same_credentials test_partition/test.py::test_system_detached_parts [gw2] [ 32%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_same_keys [gw4] [ 33%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy [gw7] [ 34%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards [gw0] [ 35%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw0] [ 36%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] [gw7] [ 37%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] [gw0] [ 38%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] test_reload_client_certificate/test.py::test_correct_cn_cert [gw4] [ 39%] PASSED test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_s3_style_link/test.py::test_s3_table_functions [gw3] [ 40%] PASSED test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options [gw0] [ 41%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] [gw7] [ 42%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] [gw1] [ 43%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] [gw0] [ 44%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw4] [ 45%] PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works [gw5] [ 46%] PASSED test_s3_style_link/test.py::test_s3_table_functions test_s3_style_link/test.py::test_s3_table_functions_line_as_string [gw0] [ 47%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw1] [ 48%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] [gw5] [ 49%] PASSED test_s3_style_link/test.py::test_s3_table_functions_line_as_string [gw6] [ 50%] PASSED test_partition/test.py::test_system_detached_parts [gw7] [ 51%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] [gw1] [ 52%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] [gw4] [ 53%] PASSED test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk [gw3] [ 54%] PASSED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_s3_table_functions/test.py::test_s3_table_functions test_read_only_table/test.py::test_restart_zookeeper test_part_uuid/test.py::test_part_uuid [gw1] [ 55%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] [gw4] [ 56%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy [gw8] [ 57%] PASSED test_recovery_replica/test.py::test_update_metadata [gw9] [ 58%] PASSED test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards [gw1] [ 59%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] test_range_hashed_dictionary_types/test.py::test_range_hashed_dict [gw4] [ 60%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy [gw1] [ 61%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] [gw3] [ 62%] PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_replicated_table_attach/test.py::test_startup_with_small_bg_pool [gw6] [ 63%] PASSED test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts [gw1] [ 64%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge [gw4] [ 65%] PASSED test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy [gw2] [ 66%] PASSED test_reload_client_certificate/test.py::test_correct_cn_cert test_reload_client_certificate/test.py::test_wrong_cn_cert [gw5] [ 67%] PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict [gw1] [ 68%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter [gw7] [ 69%] PASSED test_part_uuid/test.py::test_part_uuid [gw4] [ 70%] PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk [gw6] [ 71%] PASSED test_s3_table_functions/test.py::test_s3_table_functions_timeouts test_reload_query_masking_rules/test.py::test_reload_query_masking_rules [gw3] [ 72%] PASSED test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options [gw3] [ 73%] PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options [gw3] [ 74%] PASSED test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options [gw3] [ 75%] PASSED test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options [gw3] [ 76%] PASSED test_rocksdb_options/test.py::test_valid_options test_relative_filepath/test.py::test_filepath [gw4] [ 77%] PASSED test_reloading_storage_configuration/test.py::test_update_disk [gw8] [ 78%] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned [gw7] [ 79%] PASSED test_relative_filepath/test.py::test_filepath [gw5] [ 80%] PASSED test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_s3_imds/test_session_token.py::test_credentials_from_metadata [gw1] [ 81%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system [gw0] [ 82%] PASSED test_read_only_table/test.py::test_restart_zookeeper [gw9] [ 83%] PASSED test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards test_s3_latency_log/test.py::test_latency_log test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_parallel_replicas_increase_error_count/test.py::test_increase_error_count [gw8] [ 84%] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned test_s3_storage_class/test.py::test_s3_storage_class_right [gw7] [ 85%] PASSED test_s3_imds/test_session_token.py::test_credentials_from_metadata [gw6] [ 86%] PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach [gw1] [ 87%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation [gw8] [ 88%] PASSED test_s3_storage_class/test.py::test_s3_storage_class_right test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop [gw5] [ 89%] PASSED test_s3_latency_log/test.py::test_latency_log test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] [gw9] [ 90%] PASSED test_parallel_replicas_increase_error_count/test.py::test_increase_error_count test_s3_imds/test_simple.py::test_credentials_from_metadata [gw1] [ 91%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter [gw0] [ 92%] PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop [gw1] [ 93%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system test_replicating_constants/test.py::test_different_versions [gw6] [ 94%] PASSED test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] [gw9] [ 95%] PASSED test_s3_imds/test_simple.py::test_credentials_from_metadata [gw1] [ 96%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] [gw0] [ 97%] PASSED test_replicating_constants/test.py::test_different_versions [gw1] [ 98%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] [gw2] [ 99%] PASSED test_reload_client_certificate/test.py::test_wrong_cn_cert test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race [gw2] [100%] FAILED test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race =================================== FAILURES =================================== ________________________ test_partition_move_drop_race _________________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_partition_move_drop_race(started_cluster): for node in [node1, node2, node3]: node.query(f"DROP DATABASE IF EXISTS test_db SYNC") node1.query("SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range") node1.query("SYSTEM ENABLE FAILPOINT rmt_delay_commit_part") for node in [node1, node2, node3]: node.query( """ CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') """ ) node1.query( f""" CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 """ ) for node in [node1, node2, node3]: assert node.query_with_retry( "SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl'", check_callback=lambda x: x.strip() == "1", ) table_uuid = node1.query( "SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl'" ).strip() node2.query(f"SYSTEM STOP FETCHES test_db.tbl") node3.query(f"SYSTEM STOP FETCHES test_db.tbl") prev_block_number = get_last_block_number_for_partition(node1, table_uuid, 0) exception_holder = [None] def drop_op(): try: # Wait until INSERT query allocates the block number timeout = 60 # seconds start_time = time.time() while True: current_block_number = get_last_block_number_for_partition( node1, table_uuid, 0 ) if current_block_number != prev_block_number: break if time.time() - start_time > timeout: raise TimeoutError( f"Timeout waiting for block number to change after {timeout}s. " f"Previous: {prev_block_number}, Current: {current_block_number}" ) time.sleep(0.5) node1.query(f"ALTER TABLE test_db.tbl DROP PARTITION 0") except Exception as e: exception_holder[0] = e t = threading.Thread(target=drop_op) t.start() > node1.query(f"INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c')") test_race_condition_for_replicated_merge_tree/test.py:146: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 138, stderr: Received exception from server (version 25.3.14): E Code: 394. DB::Exception: Received from 172.16.1.5:9000. DB::Exception: Insert query (for block /clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0/block-0000000000) was canceled by concurrent ALTER PARTITION or TRUNCATE. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ecd1 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c089a11 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c420b E 3. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x000000000c6ddf98 E 4. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1116: DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda1'()::operator()() const @ 0x000000002f9c44e3 E 5. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1146: DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda2'()::operator()() const @ 0x000000002f9c0d40 E 6. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1179: void DB::ZooKeeperRetriesControl::retryLoop::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&, void DB::ZooKeeperRetriesControl::retryLoop::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()>(DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&&)::'lambda'()>(DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&&, void DB::ZooKeeperRetriesControl::retryLoop::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()>(DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&&)::'lambda'()&&) @ 0x000000002f9c0855 E 7. ./src/Common/ZooKeeper/ZooKeeperRetries.h:52: DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long) @ 0x000000002f99d15d E 8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:493: DB::ReplicatedMergeTreeSinkImpl::finishDelayedChunk(std::shared_ptr const&) @ 0x000000002f983455 E 9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1257: DB::ReplicatedMergeTreeSinkImpl::onFinish() @ 0x000000002f99bb8c E 10. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000030905fbb E 11. ./build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:187: DB::ExceptionKeepingTransform::work() @ 0x00000000309053b7 E 12. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::ExecutionThreadContext::executeTask() @ 0x00000000302efa14 E 13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:296: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x00000000302d0560 E 14. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:170: DB::PipelineExecutor::executeStep(std::atomic*) @ 0x00000000302cf515 E 15. ./build_docker/./src/Processors/Executors/PushingPipelineExecutor.cpp:131: DB::PushingPipelineExecutor::finish() @ 0x0000000030300a55 E 16. ./build_docker/./src/Server/TCPHandler.cpp:1128: DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000300d9686 E 17. ./build_docker/./src/Server/TCPHandler.cpp:673: DB::TCPHandler::runImpl() @ 0x00000000300bf95a E 18. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3da E 19. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x0000000038859fcf E 20. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885ac97 E 21. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x00000000387695eb E 22. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000387631e8 E 23. asan_thread_start(void*) @ 0x000000000c676e77 E 24. ? @ 0x00007f1dc5993ac3 E 25. ? @ 0x00007f1dc5a25850 E . (QUERY_WAS_CANCELLED) E (query: INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c')) helpers/client.py:248: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- Running tests in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/test.py Running tests in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/test.py Cluster start called. is_up=False Cluster start called. is_up=False Docker networks for project roottestraceconditionforreplicatedmergetree-gw2 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestraceconditionforreplicatedmergetree-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestraceconditionforreplicatedmergetree-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestraceconditionforreplicatedmergetree-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestraceconditionforreplicatedmergetree-gw2 are DRIVER VOLUME NAME Docker volumes for project roottestraceconditionforreplicatedmergetree-gw2 are DRIVER VOLUME NAME Cleanup called Cleanup called Docker networks for project roottestraceconditionforreplicatedmergetree-gw2 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestraceconditionforreplicatedmergetree-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestraceconditionforreplicatedmergetree-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestraceconditionforreplicatedmergetree-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestraceconditionforreplicatedmergetree-gw2 are DRIVER VOLUME NAME Docker volumes for project roottestraceconditionforreplicatedmergetree-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestraceconditionforreplicatedmergetree-gw2 No running containers for project: roottestraceconditionforreplicatedmergetree-gw2 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:1 Stdout:1 Volumes pruned: 1 Volumes pruned: 1 Setup directory for instance: node1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/database Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/database Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/logs Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node2 Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/database Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/database Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/logs Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node3 Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/database Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/database Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/logs Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml pull] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml pull] Stderr: zoo2 Skipped - Image is already being pulled by zoo1 Stderr: zoo2 Skipped - Image is already being pulled by zoo1 Stderr: zoo3 Skipped - Image is already being pulled by zoo1 Stderr: zoo3 Skipped - Image is already being pulled by zoo1 Stderr: node2 Skipped - Image is already being pulled by zoo1 Stderr: node2 Skipped - Image is already being pulled by zoo1 Stderr: node3 Skipped - Image is already being pulled by zoo1 Stderr: node3 Skipped - Image is already being pulled by zoo1 Stderr: node1 Skipped - Image is already being pulled by zoo1 Stderr: node1 Skipped - Image is already being pulled by zoo1 Stderr: zoo1 Pulling Stderr: zoo1 Pulling Stderr: zoo1 Pulled Stderr: zoo1 Pulled Setup ZooKeeper Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/coordination'] Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/coordination'] Command:[docker compose --project-name roottestraceconditionforreplicatedmergetree-gw2 --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Command:[docker compose --project-name roottestraceconditionforreplicatedmergetree-gw2 --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr:time="2026-09-21T06:46:12Z" level=trace msg="Docker Desktop integration not enabled" Stderr:time="2026-09-21T06:46:12Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Creating Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Creating Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Created Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Started Stderr:time="2026-09-21T06:46:13Z" level=debug msg="otel error" error="" Stderr:time="2026-09-21T06:46:13Z" level=debug msg="otel error" error="" Stderr:time="2026-09-21T06:46:13Z" level=debug msg="otel error" error="" Stderr:time="2026-09-21T06:46:13Z" level=debug msg="otel error" error="" Wait ZooKeeper to start Wait ZooKeeper to start get_instance_ip instance_name=zoo1 get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-zoo1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.2, port:2181, use_ssl:False get_kazoo_client: zoo1, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-zoo2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.3, port:2181, use_ssl:False get_kazoo_client: zoo2, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-zoo3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.4, port:2181, use_ssl:False get_kazoo_client: zoo3, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml up -d --no-recreate') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml up -d --no-recreate] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml up -d --no-recreate] Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Started ClickHouse instance created ClickHouse instance created get_instance_ip instance_name=node1 get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.5... Waiting for ClickHouse start in node1, ip: 172.16.1.5... http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None ClickHouse node1 started ClickHouse node1 started get_instance_ip instance_name=node2 get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.1.6... Waiting for ClickHouse start in node2, ip: 172.16.1.6... http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cf4ba69c7a90de07965f27f559e0edf4af857ae5806c8edf10703d5a91f315c6/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cf4ba69c7a90de07965f27f559e0edf4af857ae5806c8edf10703d5a91f315c6/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cf4ba69c7a90de07965f27f559e0edf4af857ae5806c8edf10703d5a91f315c6/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cf4ba69c7a90de07965f27f559e0edf4af857ae5806c8edf10703d5a91f315c6/json HTTP/1.1" 200 None ClickHouse node2 started ClickHouse node2 started get_instance_ip instance_name=node3 get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.1.7... Waiting for ClickHouse start in node3, ip: 172.16.1.7... http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/8b9a0559bfe8386b4aa9ed0449b1bc186668278a2eba0e26a88a0afc1c00c990/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/8b9a0559bfe8386b4aa9ed0449b1bc186668278a2eba0e26a88a0afc1c00c990/json HTTP/1.1" 200 None ClickHouse node3 started ClickHouse node3 started ------------------------------ Captured log setup ------------------------------ 2026-09-21 06:46:02.205000 [ 644 ] INFO : Running tests in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/test.py (cluster.py:2738, start) 2026-09-21 06:46:02.205000 [ 644 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-09-21 06:46:02.222000 [ 644 ] DEBUG : Docker networks for project roottestraceconditionforreplicatedmergetree-gw2 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 06:46:02.238000 [ 644 ] DEBUG : Docker containers for project roottestraceconditionforreplicatedmergetree-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 06:46:02.256000 [ 644 ] DEBUG : Docker volumes for project roottestraceconditionforreplicatedmergetree-gw2 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 06:46:02.257000 [ 644 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 06:46:02.274000 [ 644 ] DEBUG : Docker networks for project roottestraceconditionforreplicatedmergetree-gw2 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 06:46:02.294000 [ 644 ] DEBUG : Docker containers for project roottestraceconditionforreplicatedmergetree-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 06:46:02.311000 [ 644 ] DEBUG : Docker volumes for project roottestraceconditionforreplicatedmergetree-gw2 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 06:46:02.311000 [ 644 ] DEBUG : Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 06:46:02.328000 [ 644 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 06:46:02.328000 [ 644 ] DEBUG : No running containers for project: roottestraceconditionforreplicatedmergetree-gw2 (cluster.py:879, cleanup) 2026-09-21 06:46:02.328000 [ 644 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 06:46:02.344000 [ 644 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 06:46:02.344000 [ 644 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 06:46:02.367000 [ 644 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-09-21 06:46:02.367000 [ 644 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-09-21 06:46:02.367000 [ 644 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 06:46:02.367000 [ 644 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 06:46:02.384000 [ 644 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 06:46:02.384000 [ 644 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) 2026-09-21 06:46:02.384000 [ 644 ] DEBUG : Setup directory for instance: node1 (cluster.py:2758, start) 2026-09-21 06:46:02.385000 [ 644 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-09-21 06:46:02.385000 [ 644 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-09-21 06:46:02.385000 [ 644 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-09-21 06:46:02.386000 [ 644 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-09-21 06:46:02.386000 [ 644 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/configs/config.d (cluster.py:4741, create_dir) 2026-09-21 06:46:02.387000 [ 644 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/database (cluster.py:4758, create_dir) 2026-09-21 06:46:02.387000 [ 644 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/logs (cluster.py:4769, create_dir) 2026-09-21 06:46:02.387000 [ 644 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4850, create_dir) 2026-09-21 06:46:02.388000 [ 644 ] DEBUG : Setup directory for instance: node2 (cluster.py:2758, start) 2026-09-21 06:46:02.388000 [ 644 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-09-21 06:46:02.388000 [ 644 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-09-21 06:46:02.388000 [ 644 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-09-21 06:46:02.389000 [ 644 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-09-21 06:46:02.389000 [ 644 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/configs/config.d (cluster.py:4741, create_dir) 2026-09-21 06:46:02.389000 [ 644 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/database (cluster.py:4758, create_dir) 2026-09-21 06:46:02.390000 [ 644 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/logs (cluster.py:4769, create_dir) 2026-09-21 06:46:02.390000 [ 644 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4850, create_dir) 2026-09-21 06:46:02.390000 [ 644 ] DEBUG : Setup directory for instance: node3 (cluster.py:2758, start) 2026-09-21 06:46:02.390000 [ 644 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-09-21 06:46:02.390000 [ 644 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-09-21 06:46:02.391000 [ 644 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-09-21 06:46:02.391000 [ 644 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-09-21 06:46:02.392000 [ 644 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/configs/config.d (cluster.py:4741, create_dir) 2026-09-21 06:46:02.392000 [ 644 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/database (cluster.py:4758, create_dir) 2026-09-21 06:46:02.392000 [ 644 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/logs (cluster.py:4769, create_dir) 2026-09-21 06:46:02.392000 [ 644 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4850, create_dir) 2026-09-21 06:46:02.392000 [ 644 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env (cluster.py:96, _create_env_file) 2026-09-21 06:46:02.393000 [ 644 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-09-21 06:46:02.393000 [ 644 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-09-21 06:46:02.393000 [ 644 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-09-21 06:46:02.393000 [ 644 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-09-21 06:46:02.400000 [ 644 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-09-21 06:46:02.401000 [ 644 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-09-21 06:46:12.858000 [ 644 ] DEBUG : Stderr: zoo2 Skipped - Image is already being pulled by zoo1 (cluster.py:147, run_and_check) 2026-09-21 06:46:12.858000 [ 644 ] DEBUG : Stderr: zoo3 Skipped - Image is already being pulled by zoo1 (cluster.py:147, run_and_check) 2026-09-21 06:46:12.858000 [ 644 ] DEBUG : Stderr: node2 Skipped - Image is already being pulled by zoo1 (cluster.py:147, run_and_check) 2026-09-21 06:46:12.858000 [ 644 ] DEBUG : Stderr: node3 Skipped - Image is already being pulled by zoo1 (cluster.py:147, run_and_check) 2026-09-21 06:46:12.858000 [ 644 ] DEBUG : Stderr: node1 Skipped - Image is already being pulled by zoo1 (cluster.py:147, run_and_check) 2026-09-21 06:46:12.858000 [ 644 ] DEBUG : Stderr: zoo1 Pulling (cluster.py:147, run_and_check) 2026-09-21 06:46:12.858000 [ 644 ] DEBUG : Stderr: zoo1 Pulled (cluster.py:147, run_and_check) 2026-09-21 06:46:12.858000 [ 644 ] DEBUG : Setup ZooKeeper (cluster.py:2799, start) 2026-09-21 06:46:12.859000 [ 644 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper1/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper2/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/keeper3/coordination'] (cluster.py:2800, start) 2026-09-21 06:46:12.860000 [ 644 ] DEBUG : Command:[docker compose --project-name roottestraceconditionforreplicatedmergetree-gw2 --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] (cluster.py:121, run_and_check) 2026-09-21 06:46:13.418000 [ 644 ] DEBUG : Stderr:time="2026-09-21T06:46:12Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:147, run_and_check) 2026-09-21 06:46:13.419000 [ 644 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Creating (cluster.py:147, run_and_check) 2026-09-21 06:46:13.419000 [ 644 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Created (cluster.py:147, run_and_check) 2026-09-21 06:46:13.419000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Creating (cluster.py:147, run_and_check) 2026-09-21 06:46:13.419000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Creating (cluster.py:147, run_and_check) 2026-09-21 06:46:13.419000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Creating (cluster.py:147, run_and_check) 2026-09-21 06:46:13.419000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Created (cluster.py:147, run_and_check) 2026-09-21 06:46:13.419000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Created (cluster.py:147, run_and_check) 2026-09-21 06:46:13.419000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Created (cluster.py:147, run_and_check) 2026-09-21 06:46:13.420000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Starting (cluster.py:147, run_and_check) 2026-09-21 06:46:13.420000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Starting (cluster.py:147, run_and_check) 2026-09-21 06:46:13.420000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Starting (cluster.py:147, run_and_check) 2026-09-21 06:46:13.420000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Started (cluster.py:147, run_and_check) 2026-09-21 06:46:13.420000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Started (cluster.py:147, run_and_check) 2026-09-21 06:46:13.420000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Started (cluster.py:147, run_and_check) 2026-09-21 06:46:13.420000 [ 644 ] DEBUG : Stderr:time="2026-09-21T06:46:13Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-09-21 06:46:13.420000 [ 644 ] DEBUG : Stderr:time="2026-09-21T06:46:13Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-09-21 06:46:13.420000 [ 644 ] DEBUG : Wait ZooKeeper to start (cluster.py:2436, wait_zookeeper_to_start) 2026-09-21 06:46:13.421000 [ 644 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2005, get_instance_ip) 2026-09-21 06:46:13.423000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-zoo1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:13.424000 [ 644 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.1.2, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-09-21 06:46:13.425000 [ 644 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 06:46:13.426000 [ 644 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 06:46:13.510000 [ 644 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 06:46:13.511000 [ 644 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 06:46:13.744000 [ 644 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 06:46:13.744000 [ 644 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 06:46:14.099000 [ 644 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 06:46:14.100000 [ 644 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 06:46:14.948000 [ 644 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 06:46:14.948000 [ 644 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 06:46:16.151000 [ 644 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 06:46:16.151000 [ 644 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-09-21 06:46:16.158000 [ 644 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-09-21 06:46:16.159000 [ 644 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-09-21 06:46:16.160000 [ 644 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-09-21 06:46:16.161000 [ 644 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-09-21 06:46:16.164000 [ 644 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-09-21 06:46:16.164000 [ 644 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-09-21 06:46:16.164000 [ 644 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-09-21 06:46:16.265000 [ 644 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-09-21 06:46:16.265000 [ 644 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-09-21 06:46:16.266000 [ 644 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:2005, get_instance_ip) 2026-09-21 06:46:16.268000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-zoo2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:16.269000 [ 644 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.1.3, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-09-21 06:46:16.270000 [ 644 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 06:46:16.270000 [ 644 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-09-21 06:46:16.274000 [ 644 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-09-21 06:46:16.274000 [ 644 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-09-21 06:46:16.275000 [ 644 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-09-21 06:46:16.275000 [ 644 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-09-21 06:46:16.278000 [ 644 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-09-21 06:46:16.278000 [ 644 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-09-21 06:46:16.278000 [ 644 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-09-21 06:46:16.343000 [ 644 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-09-21 06:46:16.344000 [ 644 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-09-21 06:46:16.344000 [ 644 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:2005, get_instance_ip) 2026-09-21 06:46:16.346000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-zoo3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:16.347000 [ 644 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.1.4, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-09-21 06:46:16.348000 [ 644 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 06:46:16.349000 [ 644 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-09-21 06:46:16.354000 [ 644 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-09-21 06:46:16.355000 [ 644 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-09-21 06:46:16.356000 [ 644 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-09-21 06:46:16.356000 [ 644 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-09-21 06:46:16.359000 [ 644 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-09-21 06:46:16.359000 [ 644 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-09-21 06:46:16.360000 [ 644 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-09-21 06:46:16.460000 [ 644 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-09-21 06:46:16.460000 [ 644 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-09-21 06:46:16.461000 [ 644 ] DEBUG : All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') (cluster.py:2452, wait_zookeeper_nodes_to_start) 2026-09-21 06:46:16.461000 [ 644 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml up -d --no-recreate') (cluster.py:3139, start) 2026-09-21 06:46:16.461000 [ 644 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml up -d --no-recreate] (cluster.py:121, run_and_check) 2026-09-21 06:46:16.936000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Running (cluster.py:147, run_and_check) 2026-09-21 06:46:16.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Running (cluster.py:147, run_and_check) 2026-09-21 06:46:16.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Running (cluster.py:147, run_and_check) 2026-09-21 06:46:16.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Creating (cluster.py:147, run_and_check) 2026-09-21 06:46:16.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Creating (cluster.py:147, run_and_check) 2026-09-21 06:46:16.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Creating (cluster.py:147, run_and_check) 2026-09-21 06:46:16.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Created (cluster.py:147, run_and_check) 2026-09-21 06:46:16.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Created (cluster.py:147, run_and_check) 2026-09-21 06:46:16.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Created (cluster.py:147, run_and_check) 2026-09-21 06:46:16.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Starting (cluster.py:147, run_and_check) 2026-09-21 06:46:16.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Starting (cluster.py:147, run_and_check) 2026-09-21 06:46:16.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Starting (cluster.py:147, run_and_check) 2026-09-21 06:46:16.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Started (cluster.py:147, run_and_check) 2026-09-21 06:46:16.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Started (cluster.py:147, run_and_check) 2026-09-21 06:46:16.939000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Started (cluster.py:147, run_and_check) 2026-09-21 06:46:16.939000 [ 644 ] DEBUG : ClickHouse instance created (cluster.py:3147, start) 2026-09-21 06:46:16.939000 [ 644 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2005, get_instance_ip) 2026-09-21 06:46:16.941000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:16.942000 [ 644 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2015, get_instance_global_ipv6) 2026-09-21 06:46:16.944000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:16.944000 [ 644 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.1.5... (cluster.py:3155, start) 2026-09-21 06:46:16.946000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:16.948000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.051000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.155000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.258000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.360000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.464000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.567000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.670000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.773000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.880000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:17.983000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.086000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.189000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.293000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.396000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/53b4784c787e523ef0069ee25de87818ba1ca7ece0c7209da691a3baa94ababb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.397000 [ 644 ] DEBUG : ClickHouse node1 started (cluster.py:3159, start) 2026-09-21 06:46:18.397000 [ 644 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2005, get_instance_ip) 2026-09-21 06:46:18.399000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.400000 [ 644 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2015, get_instance_global_ipv6) 2026-09-21 06:46:18.401000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.402000 [ 644 ] DEBUG : Waiting for ClickHouse start in node2, ip: 172.16.1.6... (cluster.py:3155, start) 2026-09-21 06:46:18.404000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.406000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/cf4ba69c7a90de07965f27f559e0edf4af857ae5806c8edf10703d5a91f315c6/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.509000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/cf4ba69c7a90de07965f27f559e0edf4af857ae5806c8edf10703d5a91f315c6/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.510000 [ 644 ] DEBUG : ClickHouse node2 started (cluster.py:3159, start) 2026-09-21 06:46:18.510000 [ 644 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2005, get_instance_ip) 2026-09-21 06:46:18.511000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.512000 [ 644 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2015, get_instance_global_ipv6) 2026-09-21 06:46:18.513000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.514000 [ 644 ] DEBUG : Waiting for ClickHouse start in node3, ip: 172.16.1.7... (cluster.py:3155, start) 2026-09-21 06:46:18.515000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw2-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.517000 [ 644 ] DEBUG : http://localhost:None "GET /v1.46/containers/8b9a0559bfe8386b4aa9ed0449b1bc186668278a2eba0e26a88a0afc1c00c990/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 06:46:18.518000 [ 644 ] DEBUG : ClickHouse node3 started (cluster.py:3159, start) ----------------------------- Captured stderr call ----------------------------- Executing query DROP DATABASE IF EXISTS test_db SYNC on node1 Executing query DROP DATABASE IF EXISTS test_db SYNC on node1 Executing query DROP DATABASE IF EXISTS test_db SYNC on node2 Executing query DROP DATABASE IF EXISTS test_db SYNC on node2 Executing query DROP DATABASE IF EXISTS test_db SYNC on node3 Executing query DROP DATABASE IF EXISTS test_db SYNC on node3 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range on node1 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range on node1 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_commit_part on node1 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_commit_part on node1 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node1 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node1 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node2 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node2 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node3 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node3 Executing query CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 on node1 Executing query CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 on node1 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node2 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node2 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node3 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node3 Executing query SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SYSTEM STOP FETCHES test_db.tbl on node2 Executing query SYSTEM STOP FETCHES test_db.tbl on node2 Executing query SYSTEM STOP FETCHES test_db.tbl on node3 Executing query SYSTEM STOP FETCHES test_db.tbl on node3 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c') on node1 Executing query INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c') on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query ALTER TABLE test_db.tbl DROP PARTITION 0 on node1 Executing query ALTER TABLE test_db.tbl DROP PARTITION 0 on node1 ------------------------------ Captured log call ------------------------------- 2026-09-21 06:46:18.520000 [ 644 ] DEBUG : Executing query DROP DATABASE IF EXISTS test_db SYNC on node1 (cluster.py:3648, query) 2026-09-21 06:46:18.786000 [ 644 ] DEBUG : Executing query DROP DATABASE IF EXISTS test_db SYNC on node2 (cluster.py:3648, query) 2026-09-21 06:46:19.001000 [ 644 ] DEBUG : Executing query DROP DATABASE IF EXISTS test_db SYNC on node3 (cluster.py:3648, query) 2026-09-21 06:46:19.217000 [ 644 ] DEBUG : Executing query SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range on node1 (cluster.py:3648, query) 2026-09-21 06:46:19.433000 [ 644 ] DEBUG : Executing query SYSTEM ENABLE FAILPOINT rmt_delay_commit_part on node1 (cluster.py:3648, query) 2026-09-21 06:46:19.648000 [ 644 ] DEBUG : Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node1 (cluster.py:3648, query) 2026-09-21 06:46:19.863000 [ 644 ] DEBUG : Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node2 (cluster.py:3648, query) 2026-09-21 06:46:20.080000 [ 644 ] DEBUG : Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node3 (cluster.py:3648, query) 2026-09-21 06:46:20.295000 [ 644 ] DEBUG : Executing query CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 on node1 (cluster.py:3648, query) 2026-09-21 06:46:20.661000 [ 644 ] DEBUG : Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node1 (cluster.py:3648, query) 2026-09-21 06:46:20.927000 [ 644 ] DEBUG : Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node2 (cluster.py:3648, query) 2026-09-21 06:46:21.193000 [ 644 ] DEBUG : Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node3 (cluster.py:3648, query) 2026-09-21 06:46:21.459000 [ 644 ] DEBUG : Executing query SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl' on node1 (cluster.py:3648, query) 2026-09-21 06:46:21.675000 [ 644 ] DEBUG : Executing query SYSTEM STOP FETCHES test_db.tbl on node2 (cluster.py:3648, query) 2026-09-21 06:46:21.891000 [ 644 ] DEBUG : Executing query SYSTEM STOP FETCHES test_db.tbl on node3 (cluster.py:3648, query) 2026-09-21 06:46:22.107000 [ 644 ] DEBUG : Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 (cluster.py:3648, query) 2026-09-21 06:46:22.373000 [ 644 ] DEBUG : Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 (cluster.py:3648, query) 2026-09-21 06:46:22.373000 [ 644 ] DEBUG : Executing query INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c') on node1 (cluster.py:3648, query) 2026-09-21 06:46:23.090000 [ 644 ] DEBUG : Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/6fb2236f-08a5-4ca8-8a10-3637f20be224/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 (cluster.py:3648, query) 2026-09-21 06:46:23.356000 [ 644 ] DEBUG : Executing query ALTER TABLE test_db.tbl DROP PARTITION 0 on node1 (cluster.py:3648, query) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml stop --timeout 20] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml stop --timeout 20] Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml down --volumes] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml down --volumes] Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Removed Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Removing Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Removing Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Removed Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Removed Cleanup called Cleanup called Docker networks for project roottestraceconditionforreplicatedmergetree-gw2 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestraceconditionforreplicatedmergetree-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestraceconditionforreplicatedmergetree-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestraceconditionforreplicatedmergetree-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestraceconditionforreplicatedmergetree-gw2 are DRIVER VOLUME NAME Docker volumes for project roottestraceconditionforreplicatedmergetree-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestraceconditionforreplicatedmergetree-gw2 No running containers for project: roottestraceconditionforreplicatedmergetree-gw2 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:1 Stdout:1 Volumes pruned: 1 Volumes pruned: 1 ---------------------------- Captured log teardown ----------------------------- 2026-09-21 06:46:28.112000 [ 644 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2026-09-21 06:46:29.491000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.491000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.492000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.493000 [ 644 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-09-21 06:46:29.502000 [ 644 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-09-21 06:46:29.512000 [ 644 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-09-21 06:46:29.521000 [ 644 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/.env --project-name roottestraceconditionforreplicatedmergetree-gw2 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw2/node3/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2026-09-21 06:46:29.936000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.936000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.936000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.936000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.936000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Removing (cluster.py:147, run_and_check) 2026-09-21 06:46:29.936000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.936000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Removing (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Removing (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node1-1 Removed (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node2-1 Removed (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-node3-1 Removed (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Removing (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Removing (cluster.py:147, run_and_check) 2026-09-21 06:46:29.937000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 06:46:29.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Removing (cluster.py:147, run_and_check) 2026-09-21 06:46:29.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo2-1 Removed (cluster.py:147, run_and_check) 2026-09-21 06:46:29.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo1-1 Removed (cluster.py:147, run_and_check) 2026-09-21 06:46:29.938000 [ 644 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw2-zoo3-1 Removed (cluster.py:147, run_and_check) 2026-09-21 06:46:29.938000 [ 644 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Removing (cluster.py:147, run_and_check) 2026-09-21 06:46:29.938000 [ 644 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw2_default Removed (cluster.py:147, run_and_check) 2026-09-21 06:46:29.938000 [ 644 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 06:46:29.955000 [ 644 ] DEBUG : Docker networks for project roottestraceconditionforreplicatedmergetree-gw2 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 06:46:29.972000 [ 644 ] DEBUG : Docker containers for project roottestraceconditionforreplicatedmergetree-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 06:46:29.989000 [ 644 ] DEBUG : Docker volumes for project roottestraceconditionforreplicatedmergetree-gw2 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 06:46:29.990000 [ 644 ] DEBUG : Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 06:46:30.006000 [ 644 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 06:46:30.007000 [ 644 ] DEBUG : No running containers for project: roottestraceconditionforreplicatedmergetree-gw2 (cluster.py:879, cleanup) 2026-09-21 06:46:30.007000 [ 644 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 06:46:30.024000 [ 644 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 06:46:30.025000 [ 644 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 06:46:30.049000 [ 644 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-09-21 06:46:30.050000 [ 644 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-09-21 06:46:30.050000 [ 644 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 06:46:30.050000 [ 644 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 06:46:30.067000 [ 644 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 06:46:30.067000 [ 644 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) =============================== warnings summary =============================== test_reload_client_certificate/test.py: 38 warnings /usr/local/lib/python3.10/dist-packages/kazoo/handlers/utils.py:225: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ----------------- generated report log file: parallel2_0.jsonl ----------------- ============================== slowest durations =============================== 180.19s call test_reload_client_certificate/test.py::test_wrong_cn_cert 42.91s call test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards 41.45s call test_recovery_replica/test.py::test_update_metadata 40.74s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] 37.02s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] 32.91s teardown test_read_only_table/test.py::test_restart_zookeeper 31.81s setup test_read_only_table/test.py::test_restart_zookeeper 31.24s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 27.89s call test_reload_client_certificate/test.py::test_correct_cn_cert 26.26s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] 24.11s teardown test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 23.68s setup test_recovery_replica/test.py::test_choose_source_replica 22.79s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] 22.61s teardown test_s3_style_link/test.py::test_s3_table_functions_line_as_string 22.52s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] 22.49s setup test_replication_credentials/test.py::test_credentials_and_no_credentials 22.48s setup test_reloading_storage_configuration/test.py::test_add_disk 22.06s teardown test_s3_table_functions/test.py::test_s3_table_functions_timeouts 22.05s teardown test_s3_storage_class/test.py::test_s3_storage_class_right 21.78s teardown test_s3_imds/test_simple.py::test_credentials_from_metadata 21.67s teardown test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] 21.58s teardown test_s3_latency_log/test.py::test_latency_log 21.54s teardown test_s3_imds/test_session_token.py::test_credentials_from_metadata 21.49s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] 21.15s setup test_part_uuid/test.py::test_part_uuid 20.81s setup test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards 20.72s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 20.52s call test_partition/test.py::test_system_detached_parts 20.34s setup test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 20.20s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter 19.60s call test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 19.48s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system 19.24s setup test_replicating_constants/test.py::test_different_versions 18.93s call test_s3_latency_log/test.py::test_latency_log 18.02s setup test_reload_client_certificate/test.py::test_correct_cn_cert 17.67s setup test_s3_imds/test_session_token.py::test_credentials_from_metadata 17.66s call test_read_only_table/test.py::test_restart_zookeeper 17.64s call test_rocksdb_options/test.py::test_table_invalid_options 17.26s setup test_s3_table_functions/test.py::test_s3_table_functions 17.22s teardown test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 17.11s call test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 16.80s setup test_s3_low_cardinality_right_border/test.py::test_s3_right_border 16.57s call test_rocksdb_options/test.py::test_invalid_column_family_options 16.32s setup test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race 16.12s call test_rocksdb_options/test.py::test_invalid_options 16.07s setup test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 15.75s setup test_s3_imds/test_simple.py::test_credentials_from_metadata 15.71s setup test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 15.67s setup test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] 15.62s call test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards 15.55s setup test_partition/test.py::test_attach_check_all_parts 15.49s call test_rocksdb_options/test.py::test_table_invalid_column_family_options 14.41s setup test_rocksdb_options/test.py::test_invalid_block_based_table_options 14.24s setup test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 13.68s setup test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 13.51s setup test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 13.02s setup test_s3_latency_log/test.py::test_latency_log 12.91s call test_s3_table_functions/test.py::test_s3_table_functions_timeouts 12.80s teardown test_replication_credentials/test.py::test_same_credentials 12.68s call test_rocksdb_options/test.py::test_invalid_block_based_table_options 12.29s setup test_s3_storage_class/test.py::test_s3_storage_class_right 11.16s call test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] 10.93s teardown test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 10.65s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 10.55s call test_recovery_replica/test.py::test_choose_source_replica 10.31s call test_reloading_storage_configuration/test.py::test_new_policy_works 9.85s call test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 9.63s call test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 9.35s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge 9.22s teardown test_recovery_replica/test.py::test_update_metadata 9.21s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] 9.19s call test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race 9.17s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 9.14s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter 9.07s call test_reloading_storage_configuration/test.py::test_add_disk_to_policy 9.04s call test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 8.93s call test_recovery_replica/test.py::test_recovery 8.78s teardown test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 8.61s call test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 8.54s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] 8.36s call test_reloading_storage_configuration/test.py::test_add_policy 8.35s setup test_relative_filepath/test.py::test_filepath 8.27s call test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 8.09s teardown test_partition/test.py::test_system_detached_parts 7.93s call test_reloading_storage_configuration/test.py::test_add_volume_to_policy 7.81s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] 7.70s call test_partition/test.py::test_make_clone_in_detached 7.68s teardown test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards 7.57s setup test_s3_style_link/test.py::test_s3_table_functions 7.48s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] 7.29s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] 7.26s call test_reloading_storage_configuration/test.py::test_add_disk 7.19s teardown test_part_uuid/test.py::test_part_uuid 7.09s call test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 6.97s call test_reloading_storage_configuration/test.py::test_remove_policy 6.96s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] 6.88s call test_replication_credentials/test.py::test_different_credentials 6.83s call test_reloading_storage_configuration/test.py::test_update_disk 6.74s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] 6.72s call test_part_uuid/test.py::test_part_uuid 6.59s call test_reloading_storage_configuration/test.py::test_remove_disk 6.58s call test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 6.20s teardown test_replicating_constants/test.py::test_different_versions 6.20s call test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 6.17s call test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 5.97s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] 5.91s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] 5.64s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] 5.43s teardown test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 5.17s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] 5.07s teardown test_reload_client_certificate/test.py::test_wrong_cn_cert 4.94s call test_replication_credentials/test.py::test_credentials_and_no_credentials 4.84s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] 4.73s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] 4.67s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] 4.54s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 4.52s call test_s3_low_cardinality_right_border/test.py::test_s3_right_border 4.41s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 4.27s call test_replication_credentials/test.py::test_no_credentials 4.20s call test_replication_credentials/test.py::test_same_credentials 4.14s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] 4.12s call test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 4.01s call test_partition/test.py::test_detached_part_dir_exists 3.88s teardown test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 3.87s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 3.69s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 3.67s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 3.66s teardown test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 3.57s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] 3.22s call test_partition/test.py::test_drop_detached_parts 3.21s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] 3.19s call test_partition/test.py::test_attach_check_all_parts 3.19s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 3.06s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 2.95s call test_partition/test.py::test_partition_complex 2.92s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] 2.88s teardown test_reloading_storage_configuration/test.py::test_update_disk 2.77s teardown test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 2.70s teardown test_relative_filepath/test.py::test_filepath 2.34s teardown test_rocksdb_options/test.py::test_valid_options 2.10s call test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 1.96s teardown test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race 1.91s call test_s3_imds/test_simple.py::test_credentials_from_metadata 1.87s call test_s3_imds/test_session_token.py::test_credentials_from_metadata 1.86s setup test_partition/test.py::test_system_detached_parts 1.67s teardown test_partition/test.py::test_partition_simple 1.53s call test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 1.51s call test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 1.49s setup test_partition/test.py::test_drop_detached_parts 1.48s call test_rocksdb_options/test.py::test_valid_options 1.40s call test_s3_style_link/test.py::test_s3_table_functions_line_as_string 1.33s setup test_partition/test.py::test_partition_simple 1.20s call test_s3_style_link/test.py::test_s3_table_functions 1.17s setup test_partition/test.py::test_partition_complex 1.13s setup test_replication_credentials/test.py::test_no_credentials 1.12s call test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 0.93s call test_s3_table_functions/test.py::test_s3_table_functions 0.90s call test_s3_storage_class/test.py::test_s3_storage_class_right 0.88s teardown test_partition/test.py::test_attach_check_all_parts 0.88s call test_partition/test.py::test_cannot_attach_active_part 0.87s call test_partition/test.py::test_partition_simple 0.85s setup test_partition/test.py::test_cannot_attach_active_part 0.82s setup test_replication_credentials/test.py::test_different_credentials 0.82s call test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 0.78s call test_relative_filepath/test.py::test_filepath 0.76s setup test_replication_credentials/test.py::test_same_credentials 0.63s teardown test_partition/test.py::test_drop_detached_parts 0.32s call test_replicating_constants/test.py::test_different_versions 0.27s teardown test_partition/test.py::test_partition_complex 0.27s call test_rocksdb_options/test.py::test_table_valid_options 0.27s call test_rocksdb_options/test.py::test_valid_column_family_options 0.27s call test_rocksdb_options/test.py::test_table_valid_column_family_options 0.22s teardown test_partition/test.py::test_cannot_attach_active_part 0.01s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] 0.00s teardown test_recovery_replica/test.py::test_choose_source_replica 0.00s teardown test_partition/test.py::test_make_clone_in_detached 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s setup test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s setup test_rocksdb_options/test.py::test_invalid_options 0.00s teardown test_reload_client_certificate/test.py::test_correct_cn_cert 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] 0.00s setup test_recovery_replica/test.py::test_recovery 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s teardown test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] 0.00s setup test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s setup test_reload_client_certificate/test.py::test_wrong_cn_cert 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 0.00s teardown test_replication_credentials/test.py::test_different_credentials 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s teardown test_replication_credentials/test.py::test_credentials_and_no_credentials 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] 0.00s teardown test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] 0.00s setup test_reloading_storage_configuration/test.py::test_add_policy 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s setup test_recovery_replica/test.py::test_update_metadata 0.00s setup test_reloading_storage_configuration/test.py::test_update_disk 0.00s setup test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] 0.00s teardown test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards 0.00s setup test_s3_style_link/test.py::test_s3_table_functions_line_as_string 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 0.00s setup test_partition/test.py::test_detached_part_dir_exists 0.00s teardown test_rocksdb_options/test.py::test_invalid_block_based_table_options 0.00s teardown test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] 0.00s setup test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 0.00s setup test_rocksdb_options/test.py::test_table_invalid_options 0.00s teardown test_recovery_replica/test.py::test_recovery 0.00s teardown test_s3_table_functions/test.py::test_s3_table_functions 0.00s setup test_reloading_storage_configuration/test.py::test_remove_policy 0.00s setup test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s setup test_partition/test.py::test_make_clone_in_detached 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 0.00s teardown test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s setup test_rocksdb_options/test.py::test_valid_options 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_partition/test.py::test_detached_part_dir_exists 0.00s teardown test_replication_credentials/test.py::test_no_credentials 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter 0.00s setup test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] 0.00s setup test_rocksdb_options/test.py::test_valid_column_family_options 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] 0.00s teardown test_s3_style_link/test.py::test_s3_table_functions 0.00s teardown test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_rocksdb_options/test.py::test_invalid_options 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_options 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] 0.00s setup test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system 0.00s teardown test_s3_low_cardinality_right_border/test.py::test_s3_right_border 0.00s setup test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system 0.00s setup test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge 0.00s setup test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s teardown test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 0.00s setup test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s teardown test_rocksdb_options/test.py::test_valid_column_family_options 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter 0.00s setup test_s3_table_functions/test.py::test_s3_table_functions_timeouts 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge 0.00s teardown test_reloading_storage_configuration/test.py::test_add_policy 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 0.00s teardown test_reloading_storage_configuration/test.py::test_new_policy_works =========================== short test summary info ============================ FAILED test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race PASSED test_partition/test.py::test_attach_check_all_parts PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border PASSED test_partition/test.py::test_cannot_attach_active_part PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 PASSED test_partition/test.py::test_detached_part_dir_exists PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_reloading_storage_configuration/test.py::test_add_disk PASSED test_partition/test.py::test_drop_detached_parts PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_keys PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] PASSED test_recovery_replica/test.py::test_choose_source_replica PASSED test_replication_credentials/test.py::test_different_credentials PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_partition/test.py::test_make_clone_in_detached PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces PASSED test_replication_credentials/test.py::test_no_credentials PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] PASSED test_partition/test.py::test_partition_complex PASSED test_recovery_replica/test.py::test_recovery PASSED test_rocksdb_options/test.py::test_invalid_column_family_options PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] PASSED test_partition/test.py::test_partition_simple PASSED test_replication_credentials/test.py::test_same_credentials PASSED test_replicated_merge_tree_encryption_codec/test.py::test_same_keys PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] PASSED test_reloading_storage_configuration/test.py::test_add_policy PASSED test_rocksdb_options/test.py::test_invalid_options PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy PASSED test_s3_style_link/test.py::test_s3_table_functions PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] PASSED test_s3_style_link/test.py::test_s3_table_functions_line_as_string PASSED test_partition/test.py::test_system_detached_parts PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] PASSED test_reloading_storage_configuration/test.py::test_new_policy_works PASSED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] PASSED test_reloading_storage_configuration/test.py::test_remove_disk PASSED test_recovery_replica/test.py::test_update_metadata PASSED test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options PASSED test_s3_table_functions/test.py::test_s3_table_functions PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] PASSED test_reloading_storage_configuration/test.py::test_remove_policy PASSED test_reload_client_certificate/test.py::test_correct_cn_cert PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge PASSED test_part_uuid/test.py::test_part_uuid PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy PASSED test_s3_table_functions/test.py::test_s3_table_functions_timeouts PASSED test_rocksdb_options/test.py::test_table_invalid_options PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options PASSED test_rocksdb_options/test.py::test_table_valid_options PASSED test_rocksdb_options/test.py::test_valid_column_family_options PASSED test_rocksdb_options/test.py::test_valid_options PASSED test_reloading_storage_configuration/test.py::test_update_disk PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool PASSED test_relative_filepath/test.py::test_filepath PASSED test_reload_query_masking_rules/test.py::test_reload_query_masking_rules PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter PASSED test_read_only_table/test.py::test_restart_zookeeper PASSED test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned PASSED test_s3_imds/test_session_token.py::test_credentials_from_metadata PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system PASSED test_s3_storage_class/test.py::test_s3_storage_class_right PASSED test_s3_latency_log/test.py::test_latency_log PASSED test_parallel_replicas_increase_error_count/test.py::test_increase_error_count PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter PASSED test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] PASSED test_s3_imds/test_simple.py::test_credentials_from_metadata PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system PASSED test_replicating_constants/test.py::test_different_versions PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] PASSED test_reload_client_certificate/test.py::test_wrong_cn_cert ============ 1 failed, 99 passed, 38 warnings in 318.48s (0:05:18) ============= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_vegcn5 --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1]' test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards test_part_uuid/test.py::test_part_uuid test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple test_partition/test.py::test_system_detached_parts test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_relative_filepath/test.py::test_filepath test_reload_client_certificate/test.py::test_correct_cn_cert test_reload_client_certificate/test.py::test_wrong_cn_cert test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_s3_imds/test_session_token.py::test_credentials_from_metadata test_s3_imds/test_simple.py::test_credentials_from_metadata test_s3_latency_log/test.py::test_latency_log test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 test_s3_storage_class/test.py::test_s3_storage_class_right 'test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3]' test_s3_style_link/test.py::test_s3_table_functions test_s3_style_link/test.py::test_s3_table_functions_line_as_string test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 ' returned non-zero exit status 1.